This invention relates generally to testing frameworks for software products such as web services, and more particularly to a flexible generalized testing framework for testing different web service functionality.
Proper testing of products such as web services to verify functionality requires the generation of many different test cases with appropriate input parameters and expected responses that can be compared to actual responses. To test a web service, for example, testers must send out a large number of different HTTP requests, receive responses, and compare the actual received responses with the expected response for each request. Each test case generally must be defined in a programming language with a set of input parameters to cause an HTTP client to send out a request that invokes a desired response. Testing of web service software products using traditional testing approaches generally requires hard coding of expected results into each test assertion. This makes revising test cases, composing new test cases or applying a given test case to a new system under test a complex, time consuming and labor intensive task. Frequently, predicting expected results is difficult or impossible. Since most expected values in responses are dependent upon the input parameters sent with the request, generating expected responses typically requires a mapping of input parameters and operations to test responses. This effectively hard codes test cases, requiring them to be redone to change the test cases. For many operations, values assigned by the system under test while processing a test case cannot be predicted. Thus, although there are testing frameworks available that enable one to create large numbers of test cases, their utility is limited. They are generally unable to provide expected results for each test case. As a result, a testing framework must be hard coded and static, and specific to a particular software product. Any change in the product requires a change in the testing framework. This makes a general testing framework for web service products infeasible since expected responses depend upon many variables, some of which may be unpredictable or unknown in advance of runtime. Accordingly, designing testing frameworks even for products having limited functionality is a difficult and a complex task.
Testing using general purpose simulators to simulate expected responses of software products to a plurality of general test cases is also generally infeasible for the same reasons discussed above. Furthermore, simulators tend to be complex, must be designed and constructed for particular test cases, and cannot be changed easily. Accordingly, they lack flexibility and are not useful for testing the full range of functionality of most web service products.
It is desirable to provide a general testing framework for web services that addresses the foregoing and other problems of know testing frameworks, and it is to these ends that the present invention is directed.